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(54) Representation of video im- 
ages transmitted at reduced 
frame rate 

(57) A method and apparatus for the 
representation of video images, in 
particular aerial images with a rela- 
tive movement between a pick-up 
sensor and the scene, transmitted 
at reduced frame rate. Intermediate 
frames. (n) are interpolated or extra- 
polated, to be inserted between two 



transmitted frames (0;N), by corre- 
lating the locations of a small num- 
ber of distinct corresponding refer- 
ence points (A 0 , B Q , C 0 ; A N , B N , C N ) 
in the two transmitted frames (0; 
N), said reference points being situ- 
ated in different vertical and hori- 
zontal coordinates within each of 
said two transmitted frames (0; N), 
calculating for each picture element 
in said intermediate frames (n) the 
locations of the corresponding pic- 
ture elements in each of said two 
transmitted frames and determining 
the value of the intensity for each 
picture element (Cn) of said inter- 
mediate frame by interpolation and/ 
or extrapolation. One of said two 
transmitted frames can be repre- 
sented as a still picture for a certain 
period of time followed or preceded 
by a series of reconstructed inter- 
mediate frames providing a moving 
picture. 
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SPECIFICATION 

Method for the representation of video im- 
ages or scenes, in particular aerial images 
5 transmitted at reduced frame rate 

The invention relates to a method for the 
representation of video images or scenes, in 
particular aerial images with a relative move- 
1 0 ment between a pick-up sensor and the scene 
transmitted at reduced frame rate. 

BACKGROUND OF THE INVENTION 
The task of improving and designing effici- 

1 5 ent and reliable RF systems for the noise- 
resistant transmission of moving video aerial 
scenes, especially from remotely piloted 
vehicles (RPVs) to ground stations, motivates 
an increasing interest in suitable techniques 

20 for reducing the bit rate in the digital TV 

baseband. Major candidate techniques, as re- 
viewed by Camana (1979, IEEE Spectrum 16, 
24-29) are based on transform or DPCM 
image coding, operating at the range of ap- 

25 proximately 1 to 2 bits per picture element 
(pel), transmitted either at the standard or a 
lower TV frame rate. 

As aerial scenes transmitted at standard 
frame rate, e.g. 25 frame/s, often show a 

30 largely overlapping image content of subse- 
quent frames, the use of a lower frame rate is 
capable of providing low-complex, efficient 
transmissions with a relatively high degree of 
bit rate reduction, such as by a factor of 25 or 

35 more. However, to the human eye, the frame- 
to-frame discontinuity of the displayed scene 
movement (jerkiness), usually present in 
frame-repeated images, is annoying. On the 
other hand, the scene movement during the 

40 interval between successive transmitted 

frames is largely determined, so that a priori 
knowledge exists which can be utilized for a 
regeneration of the continuous scene move- 
ment in the received image sequence. 

45 

DESCRIPTION OF THE PRIOR ART 

A method described by Hershberger and 
Vanderkolk (M.L Hershberger, R J. Vander- 
kolk "Video Image Bandwith Reduction / 

50 Compression studies for Remotely Piloted 
Vehicles", Final Tech. Rep. January 
1975 — May 1976, ASD-Tr76-26, AD- 
A039762, Hughes Aircraft Co., Culver City, 
Calif. 90230, October 1976), for example, 

55 shifts the image displayed during the time 
interval between one transmitted frame and 
the next, uniformly in accordance to the sen- 
sor movement, based on received flight data. 
The image is updated by the next transmitted 

60 frame. 

Object movements within the background 
of the image are not appearing in this case. In 
addition, there is a disadvantage that due to 
the image shift, regions uncovered by the 
65 image occur to an increasing degree and 



these gradually limit the field of vision and 
are, in each case, only refilled when the 
image of the next transmitted frame is read-in 
and appears on the display. The information 
70 required on the reception side for calculation 
of the image shift is obtained from transmitted 
data of measurements on the aircraft and 
parameters of the system for picking-up the 
image. 

75 It is an object of the invention to provide a 
method and apparatus which with the 
transmission of images, in particular aerial 
images or scenes, at reduced frame rates 
makes it possible to represent a continuous 

80 scene movement, whereby the image dis- 
played is filling the field of vision of a display 
substantially completely also during the time 
interval between two successive transmitted 
frames. 

85 

SUMMARY OF THE INVENTION 

In accordance with the present invention 
intermediate frames are constructed to be 
inserted between two transmitted frames, said 
90 reproduction starting from a small number of 
distinct corresponding reference points in the 
two transmitted frames, said reference points 
being situated in different vertical and horizon- 
tal coordinates within each of said two 
95 transmitted frames, and being determined by 
correlation of image point locations within 
said transmitted frames calculating for each 
picture element in said intermediate frames 
the locations of the corresponding picture ele- 

100 ments in each of said two transmitted frames 
and determining the value of the intensity for 
each picture element of said intermediate 
frame by interpolation and/or extrapolation 
and issuing said value and including the step 

105 of representing one of said two transmitted 
frames as a still picture over a certain period 
of time followed or preceded by a series of 
reconstructed intermediate frames providing a 
moving picture. 

110 

DESCRIPTION OF THE DRAWING 

The present invention will be more readily 
appreciated by reference to the following de- 
tailed description when read in light of the 

1 1 5 accompanying drawing in which: 

Figure 1 shows the general geometric sys- 
tem for obtaining aerial TV image sequences 
from a height H above the ground with the TV 
camera directed obliquely forward (the x- and 

1 20 x'-axes are perpendicular to the plane of the 
drawing); 

Figure 2 shows the x-y plane of the aerial 
scene (scanned ground area) projected into 
the x'-y' plane of the image area viewed in 
125 the flight direction according to Fig. 1, 

Figure 3 shows the principle of the recon- 
struction of intermediate frames, 

Figure 4 is a block diagram of an apparatus 
for carrying out the method in accordance 
1 30 with the present invention. 
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DETAILED DESCRIPTION 

In the general geometric system according 
to Fig. 1 , aerial image sequences are obtained 
5 by a preferably stabilized image sensor (cam- 
era) with an oblique depression angle moving 
over the x y object plane parallel to the y axis. 
Thereby the x y plane of the aerial scene 
(scanned ground area) is projected into the x' 

10 y' plane^of the image area as shown in Fig. 2. 
Straight lines which extend parallel to the y- 
axis in the scene being photographed are 
reproduced in the plane of the image (Fig. 2) 
as straight lines inclined to the y-axis accord- 

1 5 ing to the perspective of the reproduction. At 
the same time they represent, in each case, 
the geometric location of a picture element 
(pel), (here the path along which the pel 
moves in the image) during movement of the 

20 camera in the direction of flight (see Fig. 1). 
The dependence of the positions of a pel of 
the time, that is its shift from frame to frame 
is determined by the displacement geometry 
within the frame sequence. It can be approxi- 

25 mated in a simplified manner, as in Fig. 3, by 
straight lines. If the positions of two pels 
(reference points) are known in two successive 
frames 0 and N actually transmitted, for ex- 
ample the pels A 0 , B 0 in frame 0 and A N , B N in 

30 frame N according to Fig. 3, it is possible to 
calculate the positions for pels the rest of the 
image field, for example for the image points 
C 0 , C N , from the relationship of the distances 
between the image points by means of the 

35 Ray Ratio Law. 

The relationship between the object-equiva- 
lent image pels C 0 and C N and their coordi- 
nates x' c0 , y' co or x' CN , y' CN in the frames 0 
and N respectively and the corresponding pel 

40 C n with the coordinates x' Cn , y' Cn in a frame n 
to be regenerated are characterised by means 
of a straight connecting line, specific for this 
pel between the frames 0 and N. 
The relationship between the y' and x' 

45 coordinates of the corresponding locations C n , 
C 0 and C n/ C N in Fig. 3 can be expressed by 
the linear equations 



50 



y'c, = ^y'c, + h 2 , 



and 



55 y' c , = h 3 y'c + h 4 . 



c„ 



Qzx'c. + 94 



(D 
(2) 

(3) 
(4) 



the coefficient h, to h 4 and g, to g 4 can be 
60 calculated from the known pel locations A 0 , 
A N and B 0 , B N where h f (i = 1 , 2, 3, 4) is 
constant for each given n (that is within each 
frame) and g ; (i = 1, 2, 3, 4) is constant for 
each given value of y' Co (that is within each 
65 frame line) (see hereinafter). The chosen form 



of the algorithms represented by the equa- 
tions (1) to (4) is oriented to processing in real 
time, taking into account a line- and frame- 
sequential scan format and is thus particularly 
70 suitable to the technical realisation of the 
method. 

The algorithms are obtained on the basis of 
the elementary geometrical relations of the 
Ray Ratio Law. 

75 Once the locations C 0 and C N with the 
sample values (intensities) S co and S CN in the 
reference frames 0 and N, respectively, have 
been determined, the value S c at the present 
location C rt in the frame n can be regenerated 

80 by the interpolation 



1 

S c „ - — [(N - n)S C0 + nS c N] 
N 



(5) 



85 



where 0<n<N, or, using the sample values 
from only one frame, by the extrapolation 



90 



Sen = S c 



or 



Sen — S c 



(6a) 



(6b) 



95 Equation (5) indicates that, in the interpo- 
lated image sequence, i.e. as n increases, due 
to the progressively decreasing and increasing 
weighting factor of S co and S CN , respectively, 
the intensities of the frame 0 fade out 

100 whereas those of the frame N fade in. This 
becomes obvious when interpolating between 
non-movement-compensated scene content, 
such as objects moving against the scene 
background, or channel error effects, as de- 

1 05 scribed below. 

The extrapolation is of interest, for example, 
in picture parts with non-overlapping scene 
content between one transmitted frame and 
the next. Then the coordinate values found 

1 10 from the equations (1) to (4) would exceed 
the limit of the active image area 



1 15 



b b 

2 2 



a 

- — >x', 
2 



Co' 



(7) 



(8) 



120 



as it occurs, for example, at the top and the 
bottom of a vertically moving scene. 

In other cases an extrapolation may be 
convenient instead of the interpolation, for 

125 example when the correlation between the 
reference frames can temporarily be deter- 
mined only inaccurately. 

Object movements within the scene do not 
appear in a frame (or partialframe) sequence 

1 30 regenerated by extrapolation, while in interpo- 



GB2 1 10 046A 



fation, in general these are characterised by a 
gradual decrease in the intensity. This case, 
where the interpolation is not based on object- 
equivalent pels, corresponds in its effect to a 
fading of the positions of the moving object in 
consecutive frames transmitted. 

The relationships are similar in channel er- 
rors. Changes in the sample value of a refer- 
ence frame caused by channel error increase 
gradually in the interpolated frame sequence, 
that is within N frames, and subsequently 
fade in a corresponding manner. 

The corresponding locations of two refer- 
ence points A 0 , A N and B 0 , B N (representing 
two object-equivalent image points) in each of 
the two successive transmitted frames 0 and 
N — the minimum number of reference points 
necessary for carrying out the method — can 
be determined by means of an image correla- 
tion method or from measured data of the 
aircraft known on the receiving side. In order 
to increase the reliability of decision it is 
possible to determine the corresponding loca- 
tions for other pels with subsequent selection 
25 according to predetermined criteria (for exam- 
ple, criteria of reliability or majority). If the 
corresponding locations of the reference po- 
ints correlation are determined especially for 
the region of an object moving within the 
30 scene, then it is also possible to determine 
object equivalent pels for interpolation within 
such regions and the course of movement can 
thus be reproduced directly instead of by 
means of an increase or decrease in the 
35 intensity as described hereinbefore. 

The coefficients h t and g ; (i — 1 , 2, 3, 4) 
used in equations (1 ) to (4) are defined as 
follows. The chosen distribution of the equa- 
tions offers advantages for the course of the 
40 calculation operations in a hardware. 



10 



15 



20 



h, 



45 



h,= 



50 



55 



h 4 = 



60 g, = 



65 



N — n- 1 , 

-n*1 2 
N - n-1 , 
N(1 - 1.) 
N - n-1, 
(N-n)1 2 
N - n-1, 
1 " kiY'c, 

1 - k,y' Cn 



(9a) 



(9b) 



(9c) 



(9d) 



(10a) 



g 2 



My'c 0 - Y'J 
1 - k iY'c fl 



70 1 - k,y' CN 

93 = 



75 g 4 



1 - k,y' Cn 

My'c. - y' Cn ) 

1 - kiY'c 



(10b) 



(10c 



(10d) 



The denominator is equal in each of the 
80 equations for the g; and h f coefficients, and 
the substitutions in these equations are: 

y'a„ - y' B „ 

1, = 1 - (11a) 

85 y' Ao -y'B 0 

i2 = y'B,-y' ao ( 1 - 1 i> < 11b > 

and 

90 



k, = 



d A y'A. - d B y' BN + x' Bn - x' A 
95k 2 = d A (1 -k, y'J + k,x' A>i 
where 



(12a) 
(12b) 



100 d A = 



y*. - v. 



x' B 



105 



d 8 = 



(12c) 



(12d) 



y b„ - y Bo 



The coordinates appearing in these equa- 
tions are the known coordinates of the refer- 
ence points A 0 (x' AO , y' A0 ) and A N (x' 

AN' Y'an) 0r 

110 B 0 (x' B0 , y' B0 ) and B N (x' BN , y' BN ). 

With respect to real time processing, the 
form of the equations (1 1a)-(12d) is such that 
they require a numerical evaluation once per 
reference frame interval, the equations (9a)- 

1 15 (9d) once per frame interval, and the equa- 
tions (10a)-(10d) once per line interval. 

The values of hj are to be determined once 
per vertical period (duration of one frame) and 
the values of g; once per horizontal period 

1 20 (duration of a line) of the video signal. Via the 
equations (1) or (3) — in combination with 
equations (10a-10d) — g t is depending on hj. 
The values of 1 1 2 ; k,, k 2 ; d A and d B must be 
determined once per N vertical periods. 

1 25 An embodiment of the present invention 
which implements the calculations of the 
equations referred to above is shown in a 
block diagram in Fig. 4. 

Signal samples of a picture frame (image), 

130 each of which samples representing the inten- 
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sity of a pel are applied on input line 10. Via 
a switching device 1 2 the samples of succes- 
sive picture frames are alternatively stored in 
one of two reference frame or image stores 
5 14, 16. The switching device is actuated each 
time the samples of a full frame have been 
stored in one of the stores. 

It be assumed that the samples of the 
earlier frame 0 are stored in store 14. After 

10 actuating the switching device the next frame 
transmitted will be read in into store 16. 

By means of a correlator 1 8 for two pels 
out of frame 0, namely A 0 with the coordi- 
nates x' A0 and y' A0 and 8 0 with the coordinates 

1 5 x' 80 and y' eo the related pels A N with coordi- 
nates x' AN and y' AN and B N with coordinates 
x' BN and y' BN are detected. From these coordi- 
nates coefficients k,, k 2 and 1 1 2 are calcu- 
lated according to equations (11a, 11b) and 

20 (1 2a-1 2d) by calculating means 20. 

The position of each pel is determined by 
their coordinates n, y' Cn and x' Cn in respect of 
time and its vertical and horizontal position. 
The actual values of these coordinates are 

25 provided by a clock and control unit 22. 
For each intermediate frame 
n(n = 1,2, . . . N-1) to be interpolated, the 
coefficients h v h 2 , h 3 , h 4 are calculated by 
calculating means 24 in accordance with 

30 equations (9a-9d). Based on these coeffici- 
ents h,, h 2 , h 3 , h 4 for each vertical position 
(line) y' Cn in an intermediate frame n the 
related positions y' c0 in frame 0 and y' CN in 
frame N are calculated by calculating means 

35 26 in accordance with equations (1) and (3). 
Based on the result of this calculation by 
further calculating means 28 the coefficients 
g t , g 2 , g 3 , g 4 are calculated in accordance 
with equation (10a-10d). Based on the result 

40 of this calculation for each pel C n with a 
horizontal coordinate x' Cn the corresponding 
coordinate x' co in frame 0 is calculated by 
further calculating means 30 in accordance 
with equation (2) and the corresponding co- 

45 ordinate x' CN in frame N in accordance with 
equation (4). 

The values S co and S CN of the intensity of 
the pels C 0 (x' co ) and C N (x' CN , y' CN ) respec- 
tively determined in this way are read out of 

50 the two stores 14 and 16 respectively and 
supplied to the interpolator-/extrapolator 
means 32. By these means 32 will be tested 
whether the calculated values of the coordi- 
nates are within the limits given by the equa- 

55 tions (7) and (8) and decide thereafter 

whether the pel C n to be issued can be or is to 
be interpolated in accordance with equation 
(5) or is to be extrapolated in accordance with 
equations ((6a-6b). In the case of interpola- 
60 tion the weighting coefficients n/N and (N- 
n)/N are to be calculated. The .values of the 
intensity S n for each pel C n thus determined 
will then be supplied to output line 34 and a 
displaying and/or recording means. 
65 In the way described all intermediate frames 



n = 1 , 2 . . . N are reconstructed. The last 
frame n = N is identical with the frame stored 
in store 1 6. When this frame N is read out of 
the store 1 6 the next frame actually 

70 transmitted is to be read in into the store 1 4. 
With the frame in store 16 now being frame 0 
the cycle described above is repeated. 

The calculations are carried out by the dif- 
ferent calculating means in a manner known 

75 to those skilled in the art. 

The method described is, in principle, also 
applicable when the relationship of pels is 
described within the temporal sequence by 
means of the exact equations of the frame 

80 geometry instead of by the simpler linear 
model. 

In addition to the interpolation or extrapola- 
tion of the frame-to-frame displacement for 
the whole scene described, a corresponding 

85 interpolation or extrapolation may take place 
for predetermined areas of the frames, for 
example for objects moving in front of the 
background, for which it is then necessary to 
determine the suitable reference points inde- 

90 pendently or, in general, in order to increase 
the exactness of the reproduction and/or con- 
trast of individual parts of the image. 

The method can, fundamentally, also be 
used for other purposes with relative move- 

95 ment between a pick-up sensor and a scene. 
Depending on the number of intermediate 
frames, the projection may occur at the origi- 
nal velocity, with quick motion effect, or with 
slow motion effect. 

100 The quality of reproduction of the interme- 
diate frames inserted into the sequence of 
transmitted frames by the method described 
above is in particular depending on the accu- 
racy of the discrete frame representations, the 

105 accuracy of the mathematical picture model 
used for the reconstruction of the intermediate 
frames and the accuracy of calculations. De- 
pending on the expenditure for realisation 
there are quality restrictions appearing in a 

1 10 smoothing of steep grey value transitions (re- 
duction of the detail contrast) and edge busi- 
ness, which may affect the recognizability and 
thus the possibility to evaluate picture details. 
Evaluation of picture detail may be im- 

1 1 5 proved without substantially increasing the 
expenditure by each time displaying the first 
of two transmitted frames as a still picture 
presentation for a given period of time, said 
presentation being followed by a series of 

1 20 intermediate frames being reproduced in ac- 
cordance with the method described above. 

The term "still picture" means the repro- 
duction of a frame without changing frame 
elements. The presentation of the still picture 

1 25 is typically effected by repeating the reproduc- 
tion of the stored transmitted frame 0 with 
the frame rate of the chosen video standard. 

During the still picture presentation for a 
given period the frame is not subjected to the 

1 30 restriction mentioned above. Thus the opera- 
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tor can concentrate on details of the picture 
(e.g. details of a target) during the still picture 
representation. The following presentation of 
moving pictures has the purpose to transfer 
5 the operator with little symptoms of fatigue to 
the next transmitted frame represented again 
as a still picture. It enables him also to realize 
possible movements of objects on the back- 
ground of the image. The operator can readily 
1 0 track areas of interest, especially if there is 
large displacement between successive 
transmitted frames. In general this avoids loss 
of time for searching in the next picture for 
the corresponding area of interest. By the 

1 5 possibility to obtain detailed information dur- 
ing the still picture period or phase given 
restrictions in reproduction quality during the 
moving picture period are less important than 
in case of continuous presentation of moving 

20 pictures. 

When using this kind of representation in 
real time, the moving picture period is pre- 
sented in quick motion, since the time avail- 
able for presentation of the frames is shorter 

25 than the normal number of reconstructed in- 
termediate frames, in this case shorter than 
the time interval between two transmitted 
frames. For example, if the moving picture 
period may be 1 second out of 4 seconds 

30 chosen as interval between two successive 
transmitted frames (corresponding to a rate of 
0.25 transmitted frames per second), then the 
duration of still picture period is 3 seconds. 
In case of extremely quick motion represen- 

35 tation during the moving picture period, it 
may be advisable to display the scene move- 
ment starting with slowly increasing speed 
and terminating with slowly decreasing speed 
instead of displaying with constant speed. A 

40 slowly increasing and decreasing of the speed 
of the scene movement is subjectively more 
advantageous for the observing operator com- 
pared with an instant start and stop of the 
scene movement. The speed can be made 

45 variable according to a triangular, trigonome- 
tric (e.g. sine; cosine) or exponential or an 
polynomial time function. 

Also the duration of the still picture period 
can be variable. It is possible to realize the 

50 implementation in such a way that the opera- 
tor can optionally select still picture presenta- 
tion, e.g. if it is desired to recognize details 
more clearly. It is possible to select a still 
picture period either manually or automati- 

55 cally. An automatic selection may be done 
e.g. depending on predetermined frame ele- 
ments. If successive transmitted frames are 
sufficiently overlapping, it is possible to ex- 
tend the still picture period over the duration 

60 of a normal transmission period by ignoring 
one or more transmitted frames, i.e. operating 
with a greater frame-to-frame interval, e.g. by 
suppressing every second frame. 
The ratio of the duration of the still picture 

65 period and the duration of the moving picture 



period can be chosen differently for individual 
frame areas. In particular the still picture 
presentation may be restricted to an area of 
the frame, e.g. an area in the center of the 

70 picture, while the rest of the picture is repre- 
sented moving. 

Optionally it is possible to increase or de- 
crease the time of presentation for a whole 
scene presentation compared with the opera- 

75 tion in real time. In particular it is possible to 
extend the time for the scene presentation 
proportional to the duration of the still picture 
period, in order to display the moving picture 
period at the speed of the real time operation, 

80 i.e. without quick motion. 

CLAIMS 

1 . A method for the representation of vi- 
deo image sequences or scenes with a relative 
85 movement between a pick-up sensor and the 
scene transmitted at reduced frame rate, 
wherein the improvement consists in the re- 
production of intermediate frames to be in- 
serted between two transmitted frames, said 
90 reproduction starting from a small number of 
distinct corresponding reference points in the 
two transmitted frames, said reference points 
being situated in different vertical and horizon- 
tal coordinates within each of said two 
95 transmitted frames, and being determined by 
correlation of image point locations within 
said transmitted frames, calculating for each 
picture element in said intermediate frames 
the locations of the corresponding picture ele- 

100 ments in each of said two transmitted frames, 
and determining the value of the intensity for 
each picture element of said intermediate 
frame by interpolation and/or extrapolation 
and issuing said value, and including the step 

105 of representing one of said two transmitted 
frames as a still picture over a certain period 
of time followed or preceded by a series of 
reconstructed intermediate frames providing a 
moving picture. 

110 2. A method according to Claim 1 in 

which the interpolation is based on a model of 
a linear relationship in the corresponding loca- 
tions of the picture elements within all recon- 
structed intermediate frames. 

115 3. A method according to Claim 1 includ- 
ing the step of: 

representing the reconstructed intermediate 
frames at the beginning with slowly increasing 
speed of movement and towards the end with 
120 decreasing speed of movement. 

4. A method according to Claim 1, in 
which the length of said still picture period is 
variable. 

5. A method according to Claim 1 in 
125 which only a partial area or partial areas of 

the intermediate frames is presented as a still 
picture. 

6. An apparatus for the reproduction of 
video scenes transmitted at reduced frame 

1 30 rates comprising storage means for alterna- 



tively storing the transmitted frames, correla- 
tion means for detecting a small number of at 
least two distinctive corresponding reference 
points in said stored frames, means for calcu- 
5 lating auxiliary coefficients from the coordi- 
nates of said reference points, means calculat- 
ing the vertical and horizontal positions in the 
stored frames for each picture element in the 
intermediate frames to be reconstructed, and 

1 0 means for interpolation and/or extrapolation 
of the intensity of each picture element of said 
intermediate frames and further comprising 
means for representing one of said two 
transmitted frames as a still picture over a 

1 5 certain period of time followed or preceded by 
a series of reconstructed intermediate frames 
providing a moving picture. 

7. A method for the representation of vi- 
deo image sequences or scenes with a relative 

20 movement between a pick-up sensor and the 
scene transmitted at reduced frame rate sub- 
stantially as described herein with reference to 
the accompanying drawings. 

8. Apparatus for the reproduction of video 
25 scenes transmitted at reduced frame rates 

substantially as described herein with refer- 
ence to Fig. 4. 
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